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Abstract This paper presents a concrete implementation of the feasible sec¬ 
ond order bundle algorithm for nonsmooth, nonconvex optimization problems 
with inequality constraints It computes the search direction by solving 
a convex quadratically constrained quadratic program. Furthermore, certain 
versions of the search direction problem are discussed and the applicability of 
this approach is justified numerically by using different solvers for the com¬ 
putation of the search direction. Finally, the good performance of the second 
order bundle algorithm is demonstrated by comparison with test results of 
other solvers on examples of the Hock-Schittkowski collection, on custom ex¬ 
amples that arise in the context of finding exclusion boxes for quadratic con¬ 
straint satisfaction problems, and on higher dimensional piecewise quadratic 
examples. 
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1 Introduction 

Nonsmooth optimization addresses to solve the optimization problem 

min fix) , . 

1 

s.t. Fi{x) <0 for alH = 1 ,..., m , 

where f,Fi : R" —R are locally Lipschitz continuous. Since Fi{x) < 0 for 
alH = 1,..., m if and only if F[x) := maxi=i_...^m CiFiix) < 0 with constants 
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Ci > 0 and since F is still locally Lipschitz continuous (cf., e.g., Mifflin 
[ 2 ^ p. 969, Theorem 6 (a)]), we can always assume m = 1 in ([1]). Therefore 
w.l.o.g. we always consider the nonsmooth optimization problem with a single 
nonsmooth constraint 


min/(a;) 
s.t. F{x) < 0 , 

where F : K” —)■ R is locally Lipschitz continuous. 

Since locally Lipschitz continuous functions are differentiable almost ev¬ 
erywhere, both / and F may have kinks and therefore already the attempt to 
solve an unconstrained nonsmooth optimization problem by a smooth solver 
(e.g., by a line search algorithm or by a trust region method) by just replacing 
the gradient by a subgradient, fails in general (cf., e.g., ZoWE [^, p. 461-462]): 
If g is an element of the subdifferential df{x), then the search direction —g 
does not need to be a direction of descent (contrary to the behavior of the 
gradient of a differentiable function). Furthermore, it can happen that {ife} 
converges towards a minimizer x, although the sequence of gradients {S/ f(xk)} 
does not converge towards 0 and therefore we cannot identify x as a minimizer. 
Moreover, it can happen that {ife} converges towards a point x, but x is not 
stationary for /. The reason for these problems is that if / is not differentiable 
at X, then the gradient V/ is discontinuous at x and therefore V/(a;) does not 
give any information about the behavior of V/ in a neighborhood of x. 

Not surprisingly, like in smooth optimization, the presence of constraints 
adds additional complexity, since constructing a descent sequence whose limit 
satisfies the constraints is (both theoretically and numerically) much more 
difficult than achieving this aim without the requirement of satisfying any 
restrictions. 

Linearly constrained nonsmooth optimization. There exist various types 
of nonsmooth solvers like, e.g., the R-algorithm by Shor [s^ or stochastic 
^orithms that try to approximate the subdifferential (e.g., by Burke et al. 
[ 5 II) or bundle algorithms which force a descent of the objective function by 
using local knowledge of the function. We will concentrate on the latter ones 
as they proved to be quite efficient. 

One of the few publicly available bundle methods is the bundle-Newton 
method for nonsmooth, nonconvex unconstrained minimization by LuKSAN 
& Vlcek [2lj. We sum up its key features: It is the only method which we 
know of that uses second order information of the objective function, which 
results in faster convergence (in particular it was shown in LuKSAN & Vlcek 
[U p. 385, Section 4] that the bundle-Newton method converges superlinearly 
for strongly convex functions). Furthermore, the search direction is computed 
by solving a convex quadratic program (QP) (based on an SQP-approach in 
some sense) and it uses a line search concept for deciding whether a serious 
step or a null step is performed. Moreover, its implementation PNEW, which 
is described in LuKSAN & Vlcek [^, is written in FORTRAN. Therefore, we 
can use the bundle-Newton method for solving linearly constrained nonsmooth 
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optimization problems (as the linear constraints can just be inserted into the 
QP without any additional difficulties). 

In general, every nonsmooth solver for unconstrained optimization can 
treat constrained problems via penalty functions. Nevertheless, choosing the 
penalty parameter well is a highly nontrivial task. Furthermore, if an appli¬ 
cation only allows the nonsmooth solver to perform a few steps (as, e.g., in 
Fendl et al. @), we need to achieve a feasible descent within these steps. 

Nonlinearly constrained nonsmooth optimization. Therefore, Fendl 
& SCHICHL 0 give an extension of the bundle-Newton method to the con¬ 
strained case in a very special way: We use second order information of the 
constraint (cf. ([2])). Furthermore, we use the SQP-approach of the bundle- 
Newton method for computing the search direction for the constrained case 
and combine it with the idea of quadratic constraint approximation, as it is 
used, e.g., in the sequential quadratically constrained quadratic programming 
method by SOLODOV (this method is not a bundle method), in the hope 
to obtain good feasible iterates, where we only accept strictly feasible points 
as serious steps. Therefore, we have to solve a strictly feasible convex QCQP 
for computing the search direction. Using such a QCQP for computing the 
search direction yields a line search condition for accepting infeasible points 
as trial points (which is different to that in, e.g., Mifflin [1^). One of the 
most important properties of the convex QP (that is used to determine the 
search direction) with respect to a bundle method is its strong duality (e.g., 
for a meaningful termination criterion, for global convergence,...) which is 
also true in the case of strictly feasible convex QCQPs (cf. Fendl & SCHiCHL 
0 ). Since there exist only a few solvers specialized in solving QCQPs (all 
written in MATLAB or C, none in FORTRAN), the method is implemented 
in MATLAB as well as in C. 

For a detailed description of the presented issues we refer the reader to 
Fendl Q- 

The paper is organized as follows: In Section [2] we give a brief description 
of the implemented variant of the second order bundle algorithm. In Section [3] 
we discuss some aspects that arise when using a convex QCQP for the compu¬ 
tation of the search direction problem like the reduction of its dimension and 
the existence of a strictly feasible starting point for its SOCP-reformulation. 
Furthermore, we justify the approach for determining the search direction by 
solving a QCQP numerically by comparing the results of some well-known 
solvers for our search direction problem. In Section |T] we provide numerical 
results for our second order bundle algorithm for some examples of the Hock- 
Schittkowski collection by Schittkowski (H, [3, for custom examples that 
arise in the context of finding exclusion boxes for a quadratic CSP (constraint 
satisfaction problem) in GloptLab by Domes Q as well as for higher dimen¬ 
sional piecewise quadratic examples, and finally we compare these results to 
those of MPBNGC by Makela 0 and SolvOpt by Kappel & Kuntse- 
VICH 0 to emphasize the good performance of the algorithm on constrained 
problems. 
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Throughout the paper we use the following notation: We denote the non¬ 
negative real numbers by ]R>o := {a: € K. : x > 0}, and the space of all 
symmetric n x n-matrices by ■ For x G K." we denote the Euclidean norm 
of x by |x|, for 1 < j < j < n we define the (MATLAB-like) colon operator 
Xi-.j '■= (xi, ... Xj), and for A G Sym(n) we denote the spectral norm of A by 


|A|. 


2 Presentation of the algorithm 

In the following section we give a brief exposition of our implemented variant 
of the second order bundle algorithm whose theoretical convergence properties 


are proved in Fendl & SCHiCHL [^. For this purpose we assume that the 
functions f,F : R" — > R are locally Lipschitz continuous, that gj G df{yj) 


and gj G dF{yj), and Gj G d'^f{yj), Gj G d‘^F{yj), where the set d‘^f{x) C 
R"^ of the substitutes for the Hessian of / at a; is defined by 



{G} if the Hessian G of / at x exists 
R"^ otherwise , 


i.e., we calculate elements of the sets d‘^f{y) and d^F(jj) (in the proof of 


convergence in 0 only approximations were required). We consider the non¬ 
smooth optimization problem Q which has a single nonsmooth constraint. 
Then the second order bundle algorithm (described in Algorithm [T|) tries to 
solve optimization problem ([2]) according to the following scheme: After choos¬ 


ing a starting point xi G R" and setting up a few positive definite matrices, we 
compute the localized approximation errors. Then we solve a convex QCQP 
to determine the search direction, where the intention of the usage of the 
quadratic constraints of the QCQP is to obtain preferably feasible points that 
yield a good descent. Therefore, we only use quadratic terms in the QCQP 
for the approximation of the constraint, but not for th e ap proximation of the 
objective function (in contrast to Fendl & Schichl [10|) to balance the ef¬ 
fort of solving the QCQP with the higher number of iterations caused by this 
simplification (in Subsection 13.11 we will even discuss a further reduction of 
the size of the QCQP). Now, after computing the aggregated data and the 
predicted descent as well as testing the termination criterion, we perform a 
line search (s. Algorithm [2]) on the ray given by the search direction which 
yields a trial point yk+i that has the following property: Either yk+i is strictly 
feasible and the objective function achieves sufficient descent (serious step) or 
yk+i is strictly feasible and the model of the objective function changes suffi¬ 
ciently (null step with respect to the objective function) or yk+i is not strictly 
feasible and the model of the constraint changes sufficiently (null step with 
respect to the constraint). Afterwards we update the iteration point Xk+i and 
the information which is stored in the bundle. Now, we repeat this procedure 
until the termination criterion is satisfied. 


Algorithm 1 0. Initialization: 
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Choose the following parameters, which will not be changed during the al¬ 
gorithm: 


Table 1 ; Initial parameters 


General 

Default 

Description 

xi e K” 


Strictly feasible initial point 

yi = xi 


Initial trial point 

e > 0 


Final optimality tolerance 

M > 2 

M = n 3 

Maximal bundle dimension 

to € (0,1) 

to = 0.001 

Initial lower bound for step size 
of serious step in line search 

to € (0, 1) 

to = 0.001 

Scaling parameter for to 

mr € (0, i) 

mp = 0.01 

Descent parameter for serious step in line search 

€ (ttil, 1) 

mp = 0.5 

Parameter for change of model of objective function 
for short serious and null steps in line search 

mp € (0,1) 

mp = 0.01 

Parameter for change of model of constraint 
for short serious and null steps in line search 

Ce(0,i) 

C = 0.01 

Coefficient for interpolation in line search 

t? > 1 

= 1 

Exponent for interpolation in line search 

Cs > 0 

Cs = 10^° 

Upper bound of the distance between Xk and yk 

Gg > 0 

Gg = 10^° 

Upper bound of the norm of the damped 
matrices {pjGj} (\pjGj \ < Gg) 

Gg > 0 

II 

Upper bound of the norm of the damped 



matrices {pjGj} (\pjGj \ < Gg) 

Gg > 0 

II 

Upper bound of the norm of the matrices 
and {6'=} Cmax(|4^|, |6^'|) < ^g) 

tp > 0 

ip = 3 

Selection parameter for pk+i 

^ 0 


Matrix selection parameter 

ir > 0 


Bundle reset parameter 

71 > 0 

7i = 1 

Coefficient for locality measure for objective function 

72 > 0 

72 = 1 

Coefficient for locality measure for constraint 

Wi > 1 

= 2 

Exponent for locality measure for objective function 

W2 > 1 

W2 = 2 

Exponent for locality measure for constraint 


Set the initial values of the data which gets changed during the algorithm: 


in = 0 {ff subsequent null and short steps) 
*s = 0 (# subsequent serious steps) 

Ji = {1} (set of bundle indices) . 
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Compute the following information at the initial trial point 

fp = fi = fiyi) 

ffp = 5i = 5(2/1) e df{yi) 

Gl = G, = G(yi) e 9V(5i) 

Fp = Fi = F{yi) < 0 (yi is strictly feasible according to assumption) 
5p =5i =5(51) e dF{yi) 

Gl = Gi = G(yi) e d^F{y,) 

and set 

Sp = Sp = s) = 0 (locality measure) 

51 = pi = 1 (damping parameter) 

= 1 (Lagrange multiplier for optimality condition) 
fc = 1 (iterator) . 

1. Determination of the matrices for the QCQP: 
if (step k — 1 and k — 2 were serious steps) A 

W = Gk + R^Gk 
else 

W = G'f + r’^GI 

end 

^ im 

Wp = “positive definite modification of fL” 
else 

^ wfc-i 

r r p r p 

end 

Compute 


— 1 V 'ig Zf J 

bundle reset 


(G^, G^) = “positive definite modification of (Gp, Gj)” for all j G Jk ■ 

(3) 


2 . Computation of the localized approximation errors: 

a') :=max(|/(xfc) -/j'l, 7i(Sj )‘^') , ^p := max (|/(xfe) -/p |, 7i(Sp)“0 
A') := max (|F(a;fe) - F/|, 72(5)')'^") , := max (|-F(a;fe) - Fp^'|, 72(3^)“^") . 
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3. Determination of the search direction: Compute the solution {dk,Vk) £ 
]gn+i ffig (convex) QCQP 

mint) + (jd^Wpd , 

^,•0 ^ 

s.t. — a!) + d^g’) < V for j G Jk 

- ttp + d^gp <v ifis <ir (4) 

F{xk) - A) + d'^g') + \dFd]d < 0 for j G Jk 
F{xk) -A’) + dFgl + \dFd^d <0 if t, < V 

and its corresponding Lagrange multiplier (A^”, Ap,/tp) G and 

set Hk := {W^ + and := d’f + 

if > 0 

else 

K^«p) = o 

end 

'tf ^ F 

is = 0 (bundle reset) 

end 

f. Aggregation: We set for the aggregation of information of the objective 
function 


(/p^5p^G^+^5^) = E XUf^,g^,PjG„s^) + X^p{f^,g^p,G^p,s^p) 

j&Jk 

a^=max(|/(xfe)-/p"|,7i(str) 


and for the aggregation of information of the constraint 


iPr 


9p, 




') = 
oJ 


E 

jeJk 




,9j, 


p,G„s';) + n';{F^,g^p, 


nk -k\ 

^p-> ^p) 


Al = max(|F(xfc) -Fj^'|,72(Sp)“") 


and we set 

Vk = -dlW^dk - 

wk = ^\Hkigp + K 


i4G>) + m^G'=)4 -al- -k’^+^AI 

j&Jk 

( - F{xk)) . 


- 


{-F{xk)) 


5. Termination criterion: 
if Wk <e 

stop 


end 
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6. Line search: We compute step sizes 0 < < 1 and tg € (0,io] by 

using the line search described in Algorithm\^ and we set 

Xk+i = Xk + t\dk (is created strictly feasible by the line search) 

Uk+i — Xk ~\~ tf^dk 

fk+i = fivk+i) , gk+i = givk+i) e df{yk+i) , Gk+i = G{yk+i) G d'^f{yk+i) 
Fk+i = F{yk+i) , gk+i = giVk+i) G dF{yk+i) , Gk+i = G{yk+i) G d‘^F{yk+i) ■ 

7. Update: 

"tf in ^ ip 

Pk+i = min(l, 
else 

Pk+l = 0 
end 

p,+i = min(l,^) 
tt > tg (serious step) 

iji — ^ 

is =is + ^ 

else (no serious step, i.e. null or short step) 

In — In F 1 

end 

Compute the updates of the locality measure 

= Sj + \xk+i - Xk\ for j G Jfe 
Sfc+i “ — yk+i\ 

= Sp + \xk+l — Xk\ 

Sp+^ =Sp + \xk+l -Xk\ ■ 

Compute the updates for the objective function approximation 

fj''~^ = fj + gj'^{xk+i - Xk) + ^pj{xk+i - Xk)'^Gj{xk+i - Xk) for j G Jk 
fk+l ~ fk+l + Pk+li^k+l Vk+l) + Vk+l) Gk+l{Xk+l Vk+l) 

= fp + gp^{xk+l - Xk) + ^{Xk+1 - Xk)'^Gp+'^{xk+l - Xk) 

and for the constraint 

Fk+l ^ pk p ^pj[xk+i - Xk)'^Gj{xk+i - Xk) for j G Jk 

^k+l ~ F'k+l F gk-^.l{xk+l Uk+l) F 2Pk+lixk+l Vk+l) Gk+l{Xk+l ~ yk+l) 
pk + l ^ pk p i(Xfe+l - Xk)'^Gp'^^{Xk+l - Xk) . 
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Compute the updates for the subgradient of the objective funetion approxi¬ 
mation 

Pj + pjGj{xk+i - Xk) for j G Jk 
fffe+l + Pk+lGk+l{Xk+l — Vk+l) 

+Gp+^(a;fe+i - Xk) 

cjj + pjGj{xk+i - Xk) for j G Jk 
9k-\-l Pk-\-lGk-\-l{Xk-\-l Vk+l) 

9p+Gp'^Hxk+i - Xk) . 

Choose Jk+i C {k — M + 2,..., fc + 1} n {1, 2,... } with fc + 1 G Jk+i ■ 
fc = fc + 1 
Co to 1 

We extend the line search of the bundle-Newton method for nonsmooth un¬ 
constrained minimization to the constrained case in the line search described 
in Algorithm [21 Before formulating the line search in detail, we give a brief 
overview of its functionality: 

Starting with the step size t = 1, we check if the point Xk + tdk is strictly 
feasible. If so and if additionally the objective function decreases sufficiently 
in this point and t is not too small, then we take Xk + tdk as new iteration 
point in Algorithm [T] (serious step). Otherwise, if the point Xk -\-tdk is strictly 
feasible and the model of the objective function changes sufficiently, we take 
Xk + tdk as new trial point (short/null step with respect to the objective 
function). If Xk + tdk is not strictly feasible, but the model of the constraint 
changes sufficiently (in particular here the quadratic approximation of the 
constraint comes into play), we take Xk + tdk as new trial point (short/null 
step with respect to the constraint). After choosing a new step size t G [0,1] 
by interpolation, we iterate this procedure. 

Algorithm 2 0. Initialization: Choose ( G (0, ^) as well as d > 1 and set 
tr = 0 o-s well as t = tjj = I- 
1. Modification of either t^ or tjj: 

if F{xk -l-tdfc) < 0 

If f{xk + tdk) < fixk) + mLVk ■ t 
tL = t 

else if f{xk + tdk) > f{xk) + mLVk ■ t 
tu = t 


9r= 

attl = 
= 


and for the constraint 


9r= 

atXl = 
= 


end 
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else if F{xk + tdk) > 0 
tu =t 
to = totu 

end 

if tL > to 
tR = tL 

return (serious step) 
end 


2. Decision of return: 


if F{xk + tdk) < 0 

Q — F tdk) dfi^Xk F tdk) 5 G — G{xk tdk) ^ ^ f(^^k “t tdk) 
_ f min(l, for < 3 

^ (0 else 

/ = f{xk F tdk) F {tL - t)g'^dk F \p{tL - t)'^dlGdk 

(3 = max(|/(a;fe + tLdk) - f\,li\tL - t\‘^^\dkr^) 

if - f3Fdl{gF p{tL - t)Gdk) > mRVk and {t - tL)\dk\ < Gs 

tR = t 

return (short/null step: change of model of the objective function) 
end 

else if F{xk F tdk) > 0 

g = g{xk F tdk) G dF{xk + tdk) , G = G{xk + tdk) G d‘^F{xk + tdk) 

p = min(l,^) 

F = F{xk F tdk) F {tL - t)g'^dk F \p{tL - t)‘^dlGdk 
P = max{\F{xk F tLdk) - F\,"f 2 \tL - t\‘^^dk\‘^^) 

G = “positive definite modification of G” (5) 

if F{xk F tLdk) - PFdl{g + p{tL - t)Gdk) > mp ■ {-\dlGdk) 

and {t - tL)\dk\ < Gs (6) 


tR = t 

return (short/null step: change of model of the constraint) 
end 
end 


3. Interpolation: Choose t G + C,{tu — tL)^,tu — C{tu — ^i)’^]- 

4 . Loop: Go to 1 

Remark 1 Similar to the line search in the bundle-Newton method for non¬ 
smooth unconstrained minimization by LuKSAN & Vlcek [21|, we want to 
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choose a new point in the interval [t^ + C{tu — tL)^T'tu — Citu — by 

interpolation. For this purpose, we set up a polynomial p passing through 
{tL, f{xk + tLdk)) and {tjj, f{xk + tjjdk)) as well as a polynomial q passing 
through {tL, P{xk+tLdk)) and {tfj, F{xk+tudk)) ■ Now we minimize p subject 
to the constraint q{t) < 0 on [Ll + C{tu — tLY,tu — ({tu — tL^] we use 
a solution t as the new point. The degree of the polynomial should be chosen 
in a way that determining t is easy (e.g., if we choose p and q as quadratic 
polynomials, then determining t consists of solving a one-dimensional linear 
equation, a one-dimensional quadratic equation and a few case distinctions). 


3 The reduced problem 

In this section we present some issues that arise when using a convex QCQP 
for the computation of the search direction problem like the reduction of its 
dimension. Moreover, we give a numerical justification of the approach of de¬ 
termining the search direction by solving a QCQP by comparing the results 
of some well-known solvers for our search direction problem. 


3.1 Reduction of problem size 

We want to reduce the problem size of the QCQP (H|). For this purpose we 
choose as a positive definite modification of Gp and Gj := & for all j G Jk, 
i.e. we choose all matrices for the constraint approximation equal to a positive 
definite modification of an aggregated Hessian of the constraint (i.e. similar to 
the choice of Wp in the bundle-Newton method for nonsmooth unconstrained 
minimization by LuKSAN & Vlcek 0 ). For the implementation, we will 
extract linear constraints Bx < b with B G and b G K™ that may occur 

in the single nonsmooth function F : MF —>■ K (via a max-function of the rows 
Blx — bi < 0 for alH = 1 ,..., rfi) in the nonsmooth constrained optimization 
problem Q and put them directly into the search direction problem (this is the 
usual way of handling linear constraints in bundle methods). For easiness of 
exposition, we drop the p-constraints. These facts altogether yield the (convex) 


QCQP 



s.t. — ^ 

F{xk) - A) + d^c 


+ + IdFG^d < 0 for j G Jk 


for j G Jk 


( 7 ) 


ddi'.{xk F d) F bi 


for i = 1 ,..., m . 
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Furthermore, we consider the following modification of the QCQP ([7]) 
min i) + -cFW^d 

d,v,u 2 ‘ 

s.t. - < V for j G Jk 

F{xk)-A>l+<fg'y+u<Q forjGJfe 
\(fG^d < u 

Bi-Xxk + d) <bi for i = 1 ,..., m , 

which is a (convex) QCQP with only one quadratic constraint. 

Remark 2 We expect that the reduced QCQP (|S]) should be solved much faster 
than the QCQP ([7]) because of the following reasons: 

An interior point method for solving QPs/QCQPs solves a linear system 
(called the KKT-system) at each iteration which is the most time consuming 
operation, i.e. the bigger the KKT-system is, the longer the interior point 
method will need to solve the problem. 

If we solved a QP to determine the search direction (we do not do this 
because of Fendl & Schichl [Sp- 9, Remark 3.6]), we would obtain | 
linear constraints for approximating F which increases the size of the KKT- 
system by I Jfc| -|- 1 rows compared to the unconstrained case (i.e. without F). 

If we solve the QCQP ([7]) to determine the search direction, we will obtain 
— in addition to the |Jfc| -I- 1 rows which are due to the linear terms — 
I Jfej -|- 1 many n x n-blocks (i.e. (| Jfc| -I- I)n rows) which are due to the \ Jk\ + 1 
quadratic terms. Since Jk is bounded by the maximal bundle dimension M 
and if we choose, e.g., M = n -|- 3 (this is the recommended default value for 
M in the bundle-Newton method by LuKSAN & Vlcek for nonsmooth 
unconstrained minimization), then the KKT-system can become very big even 
for low dimensions. 

If we solve the reduced QCQP (|S|) to determine the search direction, we 
will obtain — in addition to the jJfej -I- I rows which are due to the linear 
terms — only one n x n-block (i.e. n rows) since we only have one quadratic 
term. Therefore, if n is not too big, we expect that solving the reduced QCQP 
should not take significantly more time than solving the corresponding QP at 
least for a good interior point method and this turns out to be true indeed 
(cf. the comparisons in Subsection 13.311 . 

So the big advantage of the reduced QCQP (O is that it has a size similar 
to that of the corresponding QP (i.e. its size is much smaller than that of the 
QCQP ([7])), but it still uses quadratic information to deal with the nonlinearity 
of K. 

Furthermore, we do not need to compute a positive definite modification 
Gj of Gj in (131), and we can replace the model change condition in (jH)) by 

F{xk + tLdk) - P + dl{g + p{tL - t)Gdk) > mp ■ {-Uk) 

and therefore we do not need to compute a positive definite modification G of 
G in dni). 
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3.2 Overview of the QCQP-solvers 

The most time-consuming part of the bundle-Newton method for nonsmooth 
unconstrained minimization by LuKSAN & Vlcek [2lj is solving a (convex) 
QP . This QP is solved by the FORTRAN solver PLQDFl described in LuKSAN 
[13 which exploits the special structure of the QP. Analogously, the most time- 
consuming part of Algorithm [1] is solving the (convex) QCQP (H]). 

For solving the QC QP ®, our implementation of Algorithm [1] can use 
MOSEK by Andersen [Ij, Andersen et al. Q (which is written in C and 
available as commercial software resp. as a trial version without any limitations 
of the problem size that may be used by an academic institution for 90 days) 
or IPOPT by Wachter [s^, Wachter & Biegler [13 (which is written in 
CH—h and freely available), where the ordering represents the performance of 
the solvers according to the tests in Mittelmann ■ 

For solving the SOCP-reformulation of the QCQP ® (cf. Fendl [3, p. 116, 
Subsection 4.3.2] for details), our implementation of Algorithm [1] can use 
MOSEK, SEDUMI by Polik [111, Sturm (which is written in MATLAB 
and freely available) SDPT3 by Toh et al. [3^ (which is written in MATLAB 
and freely available), or socp by LOBO et al. [l7| (which is written in C and 
freely available). Again, the ordering represents the performance of the solvers 
according to the tests in Mittelmann (^, except for socp which was not 
tested there. 

The comparisons in Mittelmann [ 2 ^ coincide with our own observa¬ 
tions (cf. Subsection 13.31) . 


3.3 Comparison of the QCQP-solvers 


All tests were performed on an Intel Pentium IV with 3 GHz and 1 GB RAM 
running Microsoft Windows XP and MATLAB R2010a. 

We are comparing the time for solving 50 randomly generated problems of 
the following types 


L(inear) := “QP obtained by setting = 0 in QCQP d?])” 

D(ifferent) := “QCQP d?])” 

E(qual) := “QCQP dZj) with 
R(educed) := “Reduced QCQP (jS])” , 


where we set m := \ Jk\ and we choose to = 0. 

Eor obtaining a first insight, how long the computation of the search di¬ 
rection will take, we compare the plots (based on the data from Table 2 in 
appendix B of Fendl & Schichl i) of the median solving times (in mil¬ 
liseconds) for the MOSEK QCQP-solver (□), the MOSEK SOCP-solver (0), 
SEDUMI (V), and SDPT3 (A), where we use the symbols to distinguish the 
results of the different solvers (since the only purpose of this subsection is to 
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obtain a rough estimation of the solving times of the different types of search 
direction problems, we only tested these solvers here because the MATLAB 
tools CVX by Grant & Boyd [i 3| resp. YALMIP by Lofberg [Hi offer 
an excellent interface for easily generation of the input data of the different 
search direction problems for these different solvers; the performance of socp 
resp. IPOPT is discussed in Remark [S] within the framework of using one of 
these two algorithms as the (QC)QP-solver in Algorithm [1]) . 

In Figures [T] to 0] we plot the median of the solving times for various problem 



sizes. Here we see that L and R are significantly faster than D and E. To ana¬ 
lyze the difference between the first two algorithms we magnify the results of 
L (dashed line) and R (solid line) and plot the result in Figure [5l 

Remark 3 Although Andersen P, p. 131, Section 7.2 and 7.2.1] recommends 
to rather use the MOSEK SOCP-solver than the MOSEK QCQP-solver for 
solving convex QCQPs, this does not coincide with the above results in which 
the MOSEK QCQP-solver has a significantly better performance than the 
MOSEK SOCP-solver for solving a QCQP of our shape. 

The results from Figures[THS]suggest that we will only test the MOSEK QCQP- 
solver on the reduced QCQP ([5]) in higher dimensions as this is the only com¬ 
bination that does not significantly exceed the shortest duration for solving 
the corresponding QP (which is always achieved by the MOSEK QP-solver). 
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Fig. 


3; Median solving time for n = 100 and Fig. 
m = 50 


4: Median solving time for n = 100 and 
m = 100 


Therefore, we plot in Figure [5] (based on the data from Table 3 in appendix 
B of FeNDL k. SCHICHL i) the minimal & maximal (lower and upper end of 
the vertical line) and the median (horizontal line) solving times (in millisec¬ 
onds) obtained by MOSEK for L (black) and R (grey) (from n = m = 400 
on, our computer started to swap and, consequently, we did not test higher 
dimensional problems). These results justify that we will mainly concentrate 
on the reduced QCQP (|5]) in the implementation as it is the only QCQP for 
which the solving time is competitive to that of the corresponding QP. 


4 Numerical results 


In the following section we compare the numerical results of our second order 
bundle algorithm with MPBNGC by Makela and SolvOpt by Kappel 
k Kuntsevich |]3 fo r some examples of the Hock-Schittkowski collection 
by SCHITTKOWSKI [SSj, [3^, for custom examples that arise in the context of 


finding exclusion boxes for a quadratic CSP in GloptLab by Domes 0, and 
for higher dimensional piecewise quadratic examples. 


4.1 Introduction 

There are three implementations of Algorithm [1] available: A pure MATLAB 
version (for easy understanding, modifying and testing new ideas concerning 
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m=25 m=50 m=50 m=100 

Fig. 5: Magnification of the median solving time for L and R 


the algorithm); a MATLAB version in which the main parts of the algorithm 
are split into several snbroutines, where every subroutine can either be called 
as pure MATLAB code or via a C mex-file (this is useful for partially speeding 
up the algorithm, but still keeping it simple enough for modifying and testing 
many examples of the modified code); and a pure C version (for performance), 
which is used throughout all the tests. The C mex-files and the C version require 
a BLAS/LABACK implementation (e.g., ATLAS by Whaley & Petitet 
(iH . GotoBLAS by Goto & van de Geijn 0 , or the Netlib BLAS reference 
implementation by Blackford et al. a).In the unconstrained case, all three 
versions produce the same results as the original FORTRAN bundle-Newton 
method by LuKSAN & Vlcek [21|. 

Although there exist some test collections for nonsmooth unconstrained op¬ 
timization (e.g., Luksan & Vlcek |23lll a nd nonsmooth linearly constrained 
optimization (e.g., Luksan & Vlcek |22l|: also cf. Karmitsa et al. [l^ for 
an extensive comparison of numerical results), we do not know a standardized, 
prominent test collection for nonsmooth constrained optimization. Therefore, 
a common way for testing nonsmooth constrained solvers is to take a test 
collection for smooth constrained optimization (e.g., the Hock-Schittkowski 
collection from Schittkowski (3^,13) and to treat the smooth constraints 
as one nonsmooth constraint (by using a max-function). 
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Fig. 6: Minimal, median and maximal solving time 


We will make tests for 

— Algorithm [T] (with optimality tolerance e := 10“^), where we refer to the 
linearly constrained version as “BNLC”, to the version with the QCQP ([7]) 
as “Full Alg(orithm)”, and to the version with the reduced QCQP ([5]) as 
“Red(uced) Alg(orithni)” 

— MPBNGC by Makela (with the standard termination criteria; al¬ 
though MPBNGC supports the handling of multiple nonsmooth constraints, 
we do not use this feature, since we are interested here, how well the dif¬ 
ferent solvers handle the nonsmoothness of a constraint, i.e. without ex¬ 
ploiting the knowledge of the structure of a max-function; since MPBNGC 
turned out to be very fast with respect to pure solving time for the low 
dimensional examples in the case of successful termination with a station¬ 
ary point, the number of iterations and function evaluations was chosen in 
a way that in the other case the solving times of the different algorithms 
have approximately at least the same mag nitude) 

— SolvOpt by Kappel & Kuntsevich [l4j (with the standard termination 
criteria, which are described in Kuntsevich &: Kappel 0 ) 

(we choose MPBNGC and SolvOpt for our comparisons, since both are written 
in a compiled programming language, both are publicly available, and both 
support nonconvex constraints), where we will modify the termination criteria 
slightly only in Subsection 14.41 on the following examples (the corresponding 
result tables can be found in Fendl & SCHICHL i): 

— Optimization problem ([T]) with f{x) := (cci + |)^ + {x 2 + |)^ and F{x) := 
maxFi: 2 (a:) (denoted by El) resp. F{x) := max(—^ 1 , 2 ( 2 ;),^ 3 ( 2 ;)) (denoted 
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by E2), where Fi{x) := — 1, F 2 {x) := {xi — 1)^ + {x 2 + 1)^ — 1, 

and ^ 3 ( 2 ;) := {xi — 1)^ — X 2 — I, the example from Fendl & SCHiCHL (ifli . 
p. 10, Example 3.7] (denoted by E3) and the Hock-Schittkowski collection 
(in the above sense; no problems which contain nonlinear equality con¬ 
straints; linear constraints are inserted into the search direction problem 
in Algorithm[Tl feasible starting point). This yields 58 test problems which 
we will discuss in Subsection in 

— Optimization problems as described in Fendl et al. [^, p. 9, Optimiza¬ 
tion problems (55) and (56)] (for finding exclusion boxes for CSPs; where 
the nonlinear part of these optimization problems is given by the certifi¬ 
cate from Fendl et al. [^, p. 5, Equation (35)], which we will discuss in 
Subsection n 

— Higher dimensional piecewise quadratic examples with up to 100 variables 
which we will discuss in Subsection 14.41 

All test examples will be sorted with respect to the problem dimension (begin¬ 
ning with the smallest). Furthermore, we use analytic derivative information 
for all occurring functions (Note: Implementing analytic derivative informa¬ 
tion for the certificate from Fendl et al. [^, p. 5, Equation (35)] effectively, is a 
nontrivial task) and we perform all tests on the same machine as in Subsection 

m 

We introduce the following notation for the record of the solution process 
of an algorithm (which is used in this section as well as in Fendl & Schichl 

i). 


Notation 3 We define 


N := “Dimension of the optimization problem” 
Nit := “Number of performed iterations” , 


we denote the final number of evaluations of function dependent data by 

Na := “Number of calls to (/, g, G, F, g, G)” (Algorithm [T]) 

Nb := “Number of calls to (/, 5 ,F,p)” (MPBNGC) 

Nc := “Number of calls to (/, F)” (SolvOpt) 

Ng := “Number of calls to g” (SolvOpt) 

Ng := “Number of calls to p” (SolvOpt) , 

we denote the duration of the solution process by 


ti := “Time in milliseconds” 

t 2 := “Time in milliseconds (without (QC)QP)” (only relevant for Algorithm [ 1 ]) 




Title Suppressed Due to Excessive Length 


19 


and we denote the additional algorithmic information by 

R := “Remark” (e.g., if tg is modified in Algorithm [TJ 

additional SolvOpt termination information, 
supplementary problem dependent facts,...) 
nt := “No termination” (within the given number of Nit,...) 
wm := “Wrong minimum” . 

Remark 4 In particular the percentage of the time spent in the (QC)QP in 
Algorithm [ 1 ] is given by 

ti (Algorithm 0 —t 2 (Algorithmic /q\ 

ti ('Algorithmilli ' V ) 


Pi ‘ = 


For comparing the cost of evaluating function dependent data (like, e.g., func¬ 
tion values, subgradients,...) in a preferably fair way (especially for solvers 
that use different function dependent data), we will make use of the following 
realistic “credit point system” that an optimal implementation of algorithmic 
differentiation in backward mode suggests (cf. Griewank & CORLiSS [I^ and 
SCHICHL (M [HI [3^1. 


Definition 1 Let Ja, gA and Ga resp. Fa, (jA and Ga be the number of 
function values, subgradients and (substitutes of) Hessians of the objective 
function resp. the constraint that an algorithm A used for solving a nonsmooth 
optimization problem which may have linear constraints and at most one single 
nonsmooth nonlinear constraint. Then we define the cost of these evaluations 

c(A) := Ja + 3gA + 3iV ■ Ga + nlc • {Fa + ^gA + 37V • Ga) , (10) 

where nlc = 1 if the optimization problem has a nonsmooth nonlinear con¬ 
straint, and nlc = 0 otherwise. 

Since Algorithm [T] evaluates /, g, G and F, g, G at every call that computes 
function dependent data, we obtain 


c(Algorithm [T|) = (1 -I- nlc) • Na • (1 -I- 3 -I- 37V) . 


Since MPBNGC evaluates /, g and A, g at every call that computes function 
dependent data (cf. Makela 01 ), the only difference to Algorithm [ 1 ] with 
respect to c from m is that MPBNGC uses no information of Hessians and 
hence we obtain 


c(MPBNGC) = (1 -b nlc) • Nb • (1 -b 3) . 

Since SolvOpt evaluates / and F at every call that computes function depen¬ 
dent data and only sometimes g or g (cf. Kuntsevich & Kappel 0 ) , we 
obtain 

c(SolvOpt) = (1 -b nlc) • Nc -b 3(Ng -b nlc • Ng) . 

We will visualize the performance of two algorithms A and H for s € 
{c. Nit} in Subsection 14.21 and Subsection 14.31 bv the following record-plot: 
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In this plot the abscissa is labeled by the name of the test example and the 
value of the ordinate is given by rp(s) := s{B) — s(A) (i.e. if rp(s) > 0, 
then rp(s) tells us how much better algorithm A is than algorithm B with 
respect to s for the considered example in absolute numbers; if rp(s) < 0, then 
rp(s) quantifies the advantage of algorithm B in comparison to algorithm A\ 
if rp(s) = 0, then both algorithms are equally good with respect to s). The 
scaling of the plots is chosen in a way that plots that contain the same test 
examples are comparable (although the plots may have been generated by 
results from different algorithms). 

Remark 5 All results for Algorithm [1] that are given in the tables of Fendl 
& Schichl i) were obtained by using MOSEK by Andersen et al. Q for 
determining the search direction, where we used the MOSEK QCQP-solver 
which turned out to be much faster than the MOSEK SOCP-solver again 
(as we already noticed in Remark [3]). We emphasize that in our tests there 
occurred no search direction problem which MOSEK was not able to solve. 

The results for computing the search direction in Algorithm [T] with IPOPT 
by Wachter & Biegler are practically the same with respect to Nit and 
Na. Furthermore, IPOPT was as robust and reliable as MOSEK. Nevertheless, 
IPOPT was slower than MOSEK with respect to the solving time which we 
expected as IPOPT is designed for general non-linear optimization problems, 
while MOSEK is specialized in particular for QCQPs. 

When using socp by Lobo et al. for the computation of the search 
direction in Algorithm [H the results are also practically the same with respect 
to Nit and Na — as long as socp did not fail to solve the search direction 
problem: The most successful effort of stabilizing socp was achieved by the 
following idea from SEDUMI by POLIK [2^, Sturm |^ : We added an ad¬ 
ditional termination criterion to socp as it is used in SEDUMI, if SEDUMI 
cannot achieve the desired accuracy for the duality gap (the additional ter¬ 
mination criterion is referred to as pars.bigeps in SEDUMI): If the current 
duality gap is smaller than bigeps := 10“^ and differs at most by I0“® from 
the duality gap of the last iteration, then we accept the current point as a 
solution. In our empirical experiments socp tended to be more reliable, when 
we chose certatin SOCP-dependent parameters according to Eendl 0, P- 123, 
Equation (4.57)]. We were not able to make socp more robust by improving 
the strict feasibility of the starting point by solving various linear programs 
that are obtained from the primal SOCP and the dual SOCP by exploiting the 
fact that \x\^ < |a;|j for all x € K" (lp_solve by Berkelaar et al. 0|, which 
is based on the revised simplex method and which we used for computing a 
solution of these linear programs, solved all of them easily). 

At least when we used the variant of socp which was best for our purposes 
(i.e. socp with a bigeps-termination criterion) in Algorithm [T1 then we were 
able to solve all examples that we took from the Hock-Schittkowski collection, 
while we were not able to achieve this for the other variants of socp. Further¬ 
more, many examples of the nonlinearly constrained optimization problem 
from Fendl et al. [^, p. 9, Optimization problems (55) and (56)] were not 
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solvable by Algorithm [T] when using socp for the computation of the search 
direction (even when we used the best variant of socp). 


4.2 Hock-Schittkowski Test-set 

From Table 4 in Fendl & Schichl in which the results for the Hock- 
Schittkowski collection can be found and which is the basis for all plots in this 
subsection, we draw the following conclusions: 

To compare the solving time ti for the reduced algorithm (with MOSEK 
as (QC)QP-solver) and MPBNGC, we consider 



ti(Red Alg) 

t2(Red Alg) 

Pi 

ti (MPBNGC) 

HS 

1198 

961 

0.80 

1386 

HS (*) 

902 

751 

0.83 

154 


where we make use of m and in (*) we consider only those examples for 
which MPBNGC satisfied one of its termination criteria (cf. Subsubsection 
I4.3.5|) . Hence, for those examples of the Hock-Schittkowski collection for which 
MPBNGC was able to terminate successfully, MPBNGC is faster than the 
reduced algorithm. Furthermore, we notice that the reduced algorithm spent 
at least 80% of its time in the QCQP-solver, which is mostly overhead time in 
particular for the examples with lower dimension (which most examples are) 
as MOSEK has to, e.g., set up sparse matrix structures. 

The reduced algorithm needs approximately 65% of the solving time G 
of the full algorithm. Nevertheless, SolvOpt only needs approximately 23% 
resp. 36% of the solving time G of the full algorithm resp. the reduced algo¬ 
rithm. Not surprisingly, the full algorithm spent 80% of the time for solving 
the QCQPs (like the reduced algorithm did). Since SolvOpt terminated for the 
higher dimensional examples (i.e. the 15-dimensional examples 284, 285 and 
384) with points that are not stationary, while both the full and the reduced 
algorithm were able to solve them, and since the reduced algorithm needs 
significantly less pure solving time than the full algorithm for these examples 


ex 

ti(Eull Alg) 

ti(Red Alg) 

P2 

284 

92 

46 

0.50 

285 

796 

140 

0.18 

384 

589 

125 

0.21 


where p 2 '■= ti(Fun Aig) ^ niay expect that for more difficult examples the 
performance of the reduced algorithm increases with respect to ti (cf. Subsub¬ 
section 012121 and Subsection 14.41) . 

Therefore, we will concentrate our comparison of Algorithm [T] (full and 
reduced version), MPBNGG and SolvOpt on the qualitative aspects of the 
cost c of the evaluations (solid line) and the number of iterations Nit (dashed 
line; this comparison is only meaningful for the comparison between the full 
algorithm and the reduced algorithm), where we use the two different line 
types for a better distinction of the comparisons in Figure[71 in this subsection, 
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where before making detailed comparisons of our 58 examples, we give a short 
overview of them as a reason of clarity of the presentation: This yields the 
following summary table consisting of the number of examples for which the 
reduced algorithm is better than the full algorithm, MPBNGC resp. SolvOpt 
(and vice versa) 



no termi¬ 
nation 

significantly 

better 

better 

a bit 
better 

nearly 

equal 

a bit 
better 

better significantly 

better 

(Color code: Light grey) 
Nit 

0 

Full Alg 

2 

1 

3 

51 

0 

Red Alg 

1 0 

c 

0 

4 

1 

7 

40 

1 

1 4 

(Color code: Grey) 
c 

5 

MPBNGC 

2 

3 

10 

16 

8 

Red Alg 

7 7 

(Color code: Black) 
c 

3 

SolvOpt 

3 

4 

3 

1 

31 

Red Alg 

9 4 


that is visualized in Figured 



sign, better better a bit better nearly equal a bit better 

(Full Alg 

/MPBNGC 

/SolvOpt) 


sign, better no term. 
{Red Alg) {Full Alg 

/MPBNGC 

/SolvOpt) 


Fig. 7: Hock-Schittkowski collection (summary) 

and that let us draw the following conclusions: The performances of the full 
algorithm and the reduced algorithm are quite similar. The reduced algorithm 
is superior to MPBNGC in one third of the examples, for a further third of the 
examples one of these two solvers has only small advantages over the other, the 
performance differences between the two algorithms considered can be com¬ 
pletely neglected for one quarter of the examples, and for the remaining ten 
percent of the examples MPBNGC beats the reduced algorithm clearly. The 
reduced algorithm is superior to SolvOpt in about one quarter of the exam¬ 
ples, for sixty percent of the examples one of these two solvers has only small 
advantages over the other (in most cases the reduced algorithm is the slightly 
more successful one), and in the remaining twelve percent of the examples 
SolvOpt beats the reduced algorithm clearly. 

Furthermore, only the full algorithm and the reduced algorithm solved all 
examples successfully. 

Reduced algorithm vs. Full algorithm First of all, in the full algorithm /§ is only 
modified in 11 examples (34, 43, 66, 83, 100, 113, 227, 230, 264, 285, 384), while 
in the reduced algorithm this happens in 14 examples (the additional examples 
are 284, 330, 341). In all these examples /§ is only modified a few times and 
a modification only occurs at very early iterations of the optimization process 
(cf. Fendl & Schichl [10, p. 19, Remark 3.16]). 
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From Figure 14 and Figure 15 in Fendl & Schichl @ we conclude that 
the full and the reduced algorithm produce in most of the 58 examples ap¬ 
proximately the same results — exceptions from this observation are in view 
of iterations the following 7 examples: The reduced algorithm is better in 1 
example in comparison with the full algorithm, while the full algorithm is 
significantly better in 2 examples, better in 1 example and a bit better in 3 
examples in comparison with the reduced algorithm. 

In view of costs the exceptions are given by the following 18 examples: The 
reduced algorithm is significantly better in 4 examples, better in 1 example 
(33) a bit better in 1 example in comparison with the full algorithm, while the 
full algorithm is significantly better in 4 examples, better in 1 example and a 
bit better in 7 examples in comparison with the reduced algorithm. 

Reduced algorithm vs. MPBNGC MPBNGC does not satisfy any of its ter¬ 
mination criteria for five examples (15, 20, 83, 285 and 384) within the given 
number of iterations and function evaluations. For the other 53 examples from 
Figure 16 in Fendl & SCHiCHL @ we emphasize the following ones: The re¬ 
duced algorithm is significantly better in 7 examples, better in 7 examples and 
a bit better in 8 examples in comparison with MPBNGC, while MPBNGC 
is significantly better in 2 examples, better in 3 examples and a bit better in 
10 examples in comparison with the reduced algorithm. In the remaining 16 
examples the cost of the reduced algorithm and MPBNGC is practically the 
same. 

Reduced algorithm vs. SolvOpt SolvOpt terminates for the three 15-dimensional 
examples 284, 285 and 384 with points that are not stationary. For the other 55 
examples from Figure 17 in Fendl & SCHiCHL we emphasize the following 
ones: The reduced algorithm is significantly better in 4 examples and better 
in 9 examples in comparison with SolvOpt, while SolvOpt is significantly bet¬ 
ter in 3 examples, better in 4 examples and a bit better in 3 examples in 
comparison with the reduced algorithm. Except for example 233 in which the 
cost of the reduced algorithm and SolvOpt are practically the same, in all 31 
remaining examples the reduced algorithm is a bit better than SolvOpt. 


4.3 Exclusion boxes 


4-. 3.1 Basics 


We consider the quadratic CSP 

Fix) € F 

X G X 


( 11 ) 


and we assume that a solver, which is able to solve a CSP, takes the box 
u := \u,u] C X into consideration during the solution process. Fendl et al. 
0 constructed a certificate of infeasibility /, which is a nondifferentiable and 
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nonconvex function in general, with the following property: If there exists a 
vector y with 

f{y,u,u)<0, (12) 

then the CSP dTTI) has no feasible point in u and consequently this box can 
be excluded for the rest of the solution process. Therefore, a box u for which 
m holds is called an exclusion box. 

The obvious way for finding an exclusion box for the CSP m is to mini¬ 
mize / 


TOiin f{y,u, u) 

V 

and stop the minimization if a negative function value occurs. We will give 
results for this linearly constrained optimization problem with a fixed box 
(i.e. without optimizing u and v) for dimensions between 4 and 11 in Subsub¬ 
section 14.3.31 

To find at least an exclusion box v ■= [y_,v] <Z u with y_-\- r < v, where 
r S (0, M — u) is fixed, we can try to solve 

min/(y,?;, v) 

s.t. [u -I- r,U] Cm , 

where the results for this linearly constrained optimization problem with a 
variable box (i.e. with optimizing u and v) for dimensions between 8 and 21 
are discussed in Subsubsection H73.4I 

Moreover, we can enlarge an exclusion box v by solving 

max^(?;, v) 

y,2l,v 

s-t. f{y,v,v) <S,[v,v]Cu, 

where (5 < 0 is given and '■= |(§Zi)li measures the magnitude of the 

box V, and we regard an exclusion box as sufficiently large, if the objective 
function satisfies fj,{v,v) < 10“®. The discussion of the results of this nonlin- 
early constrained optimization problem for dimension 8 can be found in in 
Subsubsection 14.3.51 

The underlying data for these nonsmooth optimization problems was ex¬ 
tracted from real CSPs that occur in GloptLab by Domes [g. Apart from u 
and V, we will concentrate on the optimization of the variables y and z due 
to the large number of tested examples (cf. Subsubsection ll?3.2|) . and since 
the additional optimization of R and S did not have much impact on the 
quality of the results which was discovered in additional empirical observa¬ 
tions, where a detailed analysis of these observations goes beyond the scope of 
this paper. Furthermore, we will make our tests for the two different choices 
T = 1 and T = \y\^ of the function T, which occurs in the denominator of 
the certificate / from Fendl et al. [^, p. 5, Equation (35)], where for the 
latter one / is only defined outside of the zero set of T which has measure 
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zero — although the convergence theory of many solvers (cf., e.g., Fendl & 
SCHICHL (liil . p. 7, 3.1 Theoretical basics]) requires that all occurring functions 
are defined on the whole space. 

Remark 6 Because SolvOpt cannot distinguish between linear and nonlinear 
constraints (cf. Kuntsevich & Kappel (ifil . p. 15]), the linear constraints of 
the linearly constrained optimization problems from Fendl et al. [^, p. 9, Op¬ 
timization problem (55) and (56)] must be formulated as nonlinear constraints 
in SolvOpt. Nevertheless, we will not include the number of these evaluations 
in the computation of the cost c from (1101) for the mentioned optimization prob¬ 
lems in Subsubsection 14.3.31 and Subsubsection 14.3.41 since these evaluations 
may be considered as easy in comparison to the evaluation of the certificate / 
from Fendl et al. [^, p. 5, Equation (35)] which is the objective function in 
these optimization problems. 


4-3.2 Overview of the results 

We compare the total time ti of the solution process, where we used the 
reduced algorithm (with MOSEK as the (QClQP-solver) in the constrained 
case: From Tables 5-8 (s. Fendl & SCHiCHL Q) we obtain 



ti(Red Alg) 

t2(Red Alg) 

Pi 

ti (MPBNGC) 

ti (SolvOpt) 

Linearly constrained (fixed box) 

1477 

215 

T = 1 
0.85 

231 

2754 

Linearly constrained (variable box) 

782 

60 

0.92 

30 

1546 

Nonlinearly constrained 

25420 

4885 

0.81 

21860 

38761 

Nonlinearly constrained (*) 

19053 

3723 

0.80 

2067 

30312 




T=\y\^ 



Linearly constrained (fixed box) 

1316 

129 

0.90 

15 

1508 

Linearly constrained (variable box) 

797 

45 

0.94 

30 

2263 

Nonlinearly constrained 

24055 

4284 

0.82 

25383 

16909 

Nonlinearly constrained (*) 

18038 

3112 

0.83 

3719 

12635 


where we make use of (jO]) and in (*) we consider only those examples for which 
MPBNGC satisfied one of its termination criteria (cf. Subsubsection 333]) . 

For the linearly constrained problems MPBNGC was the fastest of the 
tested algorithms, followed by BNLC and SolvOpt. If we consider only those 
nonlinearly constrained examples for which MPBNGC was able to terminate 
successfully, MPBNGC was the fastest algorithm again. Considering the com¬ 
petitors, for the nonlinearly constrained problems with T = \ the reduced 
algorithm is 13.3 seconds resp. 11.3 seconds faster than SolvOpt, while for 
the nonlinearly constrained problems with T = jj/j^ SolvOpt is 7.1 seconds 
resp. 5.4 seconds faster than the reduced algorithm. 

Again (cf. Subsection l4.2ll . taking a closer look at pi yields the observation 
that at least 85% of the time is consumed by solving the QP (in the linearly 
constrained case) resp. at least 80% of the time is consumed by solving the 
QCQP (in the nonlinearly constrained case), which implies that the difference 
in the percentage between the QP and the QCQP is small in particular (an 
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investigation of the behavior of the solving time ti for higher dimensional 
problems can be found in Subsection 14.41) . 

Therefore, we will concentrate in Subsubsection ITTlbdl Subsubsection H331 
and Subsubsection I4.d.5l on the comparison of qualitative aspects between 
Algorithm [3 MPBNGC and SolvOpt (like, e.g., the cost c of the evaluations), 
where before making these detailed comparisons, we give a short overview of 
them as a reason of clarity of the presentation: In both cases T = 1 (solid 
line) and T = \y\^ (dashed line), where we use the two different line types 
for a better distinction in the following, we tested 128 linearly constrained 
examples with a fixed box, 117 linearly constrained examples with a variable 
box and 201 nonlinearly constrained examples, which yields the following two 
summary tables consisting of the number of examples for which Algorithm [1] 
(BNLC resp. the reduced algorithm) is better than MPBNGC resp. SolvOpt 
(and vice versa) with respect to the cost c of the evaluations 


(Color code: Light grey) 

MPBNGC 

no termi- significantly better a bit 

nation better better 

nearly 

equal 

BNLC/Red Alg 

a bit better significantly 

better better 

Linearly constrained (fixed box) 
Linearly constrained (variable box) 
Nonlinearly constrained 

T = 1 

0 2 5 12 106 2 0 1 

0 0 0 1 116 0 0 0 

32 6 28 89 31 10 2 3 

Linearly constrained (fixed box) 
Linearly constrained (variable box) 
Nonlinearly constrained 

T = \y\2 

0 2 5 30 91 0 0 0 

0 0 0 5 112 0 0 0 

43 4 28 59 30 15 14 8 

(Color code: Black) 

SolvOpt 

no termi- significantly better a bit 

nation better better 

nearly 

equal 

BNLC/Red Alg 

a bit better significantly 

better better 

Linearly constrained (fixed box) 
Linearly constrained (variable box) 
Nonlinearly constrained 

T = 1 

0 1 3 0 61 25 13 25 

0 0 0 0 48 37 24 8 

0 0 14 20 21 76 20 50 

Linearly constrained (fixed box) 
Linearly constrained (variable box) 
Nonlinearly constrained 

T = \y\2 

0 1 2 1 32 34 49 9 

0 0 0 5 41 32 19 20 

0 2 24 26 31 61 45 12 


that are visualized in Figures [3 [H and [TUI 



sign, better better a bit better nearly equal a bit better better sign, better no term. 

(MPBNGC (BNLC) (MPBNGC 

/SolvOpt) /Sol\Opt) 


Fig. 8: Linearly constrained — fixed box (summary) 
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Fig. 9: Linearly constrained — variable box (summary) 



sign, better belter a bit better nearly equal a bit better better sign better no term. 
(MPBNGC (BNLC) (MPBNGC 

/SolvOpt) /Sol\Opt) 


Fig. 10: Nonlinearly constrained (summary) 

and that let us draw the following conclusions: 

The performance differences between BNLC and MPBNGC can be ne¬ 
glected for the largest part of the linearly constrained examples (with small 
advantages for MPBNGC in about ten percent of these examples). For the 
nonlinearly constrained examples the reduced algorithm is superior to MPB¬ 
NGC in one quarter of the examples, for forty percent of the examples one of 
these two solvers has small advantages over the other (in most cases MPBNGC 
is the slightly more successful one), the performance differences between the 
two algorithms considered can be completely neglected for fifteen percent of 
the examples, and for further fifteen percent of the examples MPBNGC beats 
the reduced algorithm clearly. 

For the linearly constrained examples BNLC is superior to SolvOpt in one 
third of the examples, for one quarter of the examples one of these two solvers 
has small advantages over the other (in nearly all cases BNLC is the slightly 
more successful one), the performance differences between the two algorithms 
considered can be completely neglected for forty percent of the examples, and 
in only one percent of the examples SolvOpt beats the reduced algorithm 
clearly. For the nonlinearly constrained examples the reduced algorithm is 
superior to SolvOpt in one third of the examples, for 45 percent of the examples 
one of these two solvers has small advantages over the other (the reduced 
algorithm is often the slightly more successful one), the performance differences 
between the considered two algorithms can be completely neglected for ten 
percent of the examples, and in the remaining ten percent of the examples 
SolvOpt beats the reduced algorithm clearly. 
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In contrast to the linearly constrained case, in which all three solvers ter¬ 
minated successfully for all examples, only the reduced algorithm and SolvOpt 
were able to attain this goal in the nonlinearly constrained case, too. 


4-3.3 Linearly constrained case (fixed hox) 

We took 310 examples from real CSPs that occur in GloptLab. We observe 
that for 79 examples the starting point is feasible for the CSP and for 103 
examples the evaluation of the certificate at the starting point identifies the 
box as infeasible and hence there remain 128 test problems. 


BNLC vs. MPBNGC In the case T = 1 we conclude from Figure 18 in Fendl 
& Schichl @ that BNLC is significantly better in 1 example and a bit better 
in 2 examples in comparison with MPBNGC, while MPBNGC is significantly 
better in 2 examples, better in 5 examples and a bit better in 12 examples in 
comparison with BNLC. In the 106 remaining examples the costs of BNLC 
and MPBNGC are practically the same. 

In the case T = \y\^ it follows from Figure 19 in Fendl & SCHiCHL @ 
that MPBNGC is significantly better in 2 examples, better in 5 examples and 
a bit better in 30 examples in comparison with BNLC. In the 91 remaining 
examples the costs of BNLC and MPBNGC are practically the same. 


BNLC vs. SolvOpt In the case T = 1 we conclude from Figure 20 in Fendl 
& Schichl Q that BNLC is significantly better in 25 examples, better in 
13 examples and a bit better in 25 examples in comparison with SolvOpt, 
while SolvOpt is significantly better in 1 example and better in 3 examples in 
comparison with BNLC. In the 61 remaining examples the costs of BNLC and 
SolvOpt are practically the same. 

In the case T = \y\^ it follows from Figure 21 in Fendl & Schichl @ 
that BNLC is significantly better in 9 examples, better in 49 examples and 
a bit better in 34 examples in comparison with SolvOpt, while SolvOpt is 
significantly better in 1 example, better in 2 examples and a bit better in 1 
example in comparison with BNLC. In the 32 remaining examples the costs 
of BNLC and SolvOpt are practically the same. 


4 . 3.4 Linearly constrained case (variable box) 

We observe that for 80 examples the starting point is feasible for the CSP 
and for 113 examples the evaluation of the certificate at the starting point 
identifies the boxes as infeasible and hence there remain 117 test problems of 
the 310 original examples from GloptLab. 
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BNLC vs. MPBNGC In the case T = 1 we conclude from Figure 22 in Fendl 
& SCHICHL @ that MPBNGC is a bit better in 1 example in comparison with 
BNLC. In the 116 remaining examples the costs of BNLC and MPBNGC are 
practically the same. 

In the case T = \y\.^ it follows from Figure 23 in Fendl & SCHiCHL Q 
that MPBNGC is a bit better in 5 examples in comparison with BNLC. In the 
112 remaining examples the costs of BNLC and MPBNGC are practically the 
same. 

BNLC vs. SolvOpt In the case T = 1 we conclude from Figure 24 in Fendl 
& SCHiCHL 0 that BNLC is significantly better in 8 examples, better in 24 
examples and a bit better in 37 examples in comparison with SolvOpt. In the 
48 remaining examples the costs of BNLC and SolvOpt are practically the 
same. 

In the case T = it follows from Figure 25 in Fendl & SCHiCHL Q 
that BNLC is significantly better in 20 examples, better in 19 examples and a 
bit better in 32 examples in comparison with SolvOpt, while SolvOpt is a bit 
better in 5 examples in comparison with BNLC. In the 41 remaining examples 
the costs of BNLC and SolvOpt are practically the same. 


4-3.5 Nonlinearly constrained case 

Since we were not able to find a starting point, i.e. an infeasible sub-box, for 
109 examples, we exclude them from the following tests for which there remain 
201 examples of the 310 original examples from GloptLab. 

Reduced algorithm vs. MPBNGC In the case T = 1 MPBNGC does not sat¬ 
isfy any of its termination criteria for 32 examples within the given number 
of iterations and function evaluations (also cf. Subsubsection 14.3.11) . For the 
remaining 169 examples we conclude from Figure 26 in Fendl & SCHiCHL Q 
that the reduced algorithm is significantly better in 3 examples, better in 2 
examples and a bit better in 10 examples in comparison with MPBNGC, while 
MPBNGC is significantly better in 6 examples, better in 28 examples and a 
bit better in 89 examples in comparison with the reduced algorithm, and in 
31 examples the costs of the reduced algorithm and MPBNGC are practically 
the same. 

In the case T = MPBNGC does not satisfy any of its termination cri¬ 
teria for 43 examples within the given number of iterations and function eval¬ 
uations. For the remaining 158 examples it follows from Figure 27 in Fendl & 
SCHiCHL that the reduced algorithm is significantly better in 8 examples, 
better in 14 examples and a bit better in 15 examples in comparison with 
MPBNGC, while MPBNGC is significantly better in 4 examples, better in 28 
examples and a bit better in 59 examples in comparison with the reduced al¬ 
gorithm, and in 30 examples the costs of the reduced algorithm and MPBNGC 
are practically the same. 





30 


Hannes Fendl, Hermann Schichl 


Reduced algorithm vs. SolvOpt In the case T = 1 we conclude from Figure 28 
in Fendl & Schichl Q that the reduced algorithm is significantly better in 50 
examples, better in 20 examples and a bit better in 76 examples in comparison 
with SolvOpt, while SolvOpt is better in 14 examples and a bit better in 
20 examples in comparison with the reduced algorithm. In the 21 remaining 
examples the costs of the reduced algorithm and SolvOpt are practically the 
same. 

In the case T = \y\.^ it follows from Figure 29 in Fendl & SCHiCHL Q 
that the reduced algorithm is significantly better in 12 examples, better in 45 
examples and a bit better in 61 examples in comparison with SolvOpt, while 
SolvOpt is significantly better in 2 examples, better in 24 examples and a 
bit better in 26 examples in comparison with the reduced algorithm. In the 
31 remaining examples the costs of the reduced algorithm and SolvOpt are 
practically the same. 


4.4 Higher dimensional piecewise quadratic examples 

We want to give numerical results for the nonsmooth optimization problem 


(P|) with 
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= max Mx) 

5 
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and Oi, j3j € K, a^, hj S Ai, Bj S Xi, Xj S 

The underlying data of the test examples was produced by a random num¬ 
ber generator with the following restrictions concerning the data corresponding 
to F: At least one Bj is chosen as a positive definite matrix to guarantee that 
the feasible set is bounded, and after choosing bj, Bj, Xj as well as a starting 
point 13j is chosen such that x^ is strictly feasible. 

We made comparison tests for the dimensions N G {20,40,60,80,100} to 
investigate the behavior of the reduced algorithm (□), MPBNGC (<0>) and 
SolvOpt (V), where we use the colors to distinguish the results of the different 
solvers, with respect to the solving time ti and successful termination, and we 
focus on the larger values of N (due to the magnitude of A^, we did not test the 
full version of Algorithm [T]). Moreover, we chose mi := ^ and m 2 G 
so that the emphasis of the examples lies on the handling of the constraint. 

Furthermore, due to the magnitude of the test examples, we weakened 
the optimality tolerance of the reduced algorithm to e := 10“^. Since the 
reduced algorithm terminated for all examples of this class of test functions 
with satisfying its termination criterion (which guarantees the stationarity of 
the computed point due to Fendl & Schichl (lOj), we denote the minimizer 
(of the corresponding example) that was computed by the reduced algorithm 
by X. 
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Before the actual tests, we performed a few runs of the whole test set, where 
we started with very weak termination criteria for MPBNGC and SolvOpt 
and then sharpened them, with the goal to make the results between the 
different solvers comparable in the following way: If the computed minimizer 
is close to X, then approximately the same Fj should be active. Based on these 
empirical observations, we made the final choices for the termination criteria 
of MPBNGC and SolvOpt, where we were quite successful to achieve this goal 
for MPBNGC, while we were not able to achieve it for SolvOpt in many cases 
(although putting a lot of effort into it). 

For every pair (V, m 2 ) we tested 20 different examples for two levels of 
difficulty that is classified by the average number of j G {!,...,m 2 } with 
\Fj{x) — F{x)\ < 10“^, which yields the following overview of our overall 400 
different examples 


Level 

m 2 

20 

40 

N 

60 

80 

100 

Easy 

N 

2 

4 

4 

6 

6 

7 


N 

5 

6 

8 

9 

10 

Difficult 

N 

2 

4 

8 

12 

15 

19 


N 

7 

14 

19 

26 

31 


i.e. for given N and m 2 we regard an example as more difficult, the more 
impact the constraint has at x (in the case of the successful termination of one 
of the solvers, there was always at least one Fj active). Moreover, for a given 
level of difficulty, N, and m 2 , the corresponding examples are sorted by the 
numbers TV — 20 + f,..., V. 

Before making detailed comparisons of the obtained results (s. Tables 9- 
12 in Fendl & SCHICHL i) in Subsubsections l4.4.1fHl^T^ we give a short 
overview of them as a reason of clarity of the presentation: For all N G 
{20,40,60,80,100} we summarize the easy examples and the difficult exam¬ 
ples, where we use two different line types for a better distinction of the com¬ 
parisons of m 2 (for m 2 = Y ^ dashed line and for m 2 = N we use 

a solid line) in Figures [H] and [T^l which yields the following two summary 
tables consisting of the number of examples for which the reduced algorithm 
is better than MPBNGC resp. SolvOpt (and vice versa) with respect to the 
solving time ti 
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that are visualized in Figure [TT] and Figure [T^ 


100 



/SolviOpt) /SolvOpt) 


Fig. 11: Easy examples (summary) 



sign better better a bit better nearly equal a bit better better sign, better no term 
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Fig. 12: Difficult examples (summary) 

and that let us together with Figure [T21 in which the solving times ti for all 
examples are plotted 



Fig. 13: Solving time tifor all higher dimensional piecewise quadratic examples 

draw the following conclusions: 

For the easy examples the reduced algorithm is superior to MPBNGC in 
thirteen percent of the examples, for thirty percent of the examples one of 
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these two solvers has small advantages over the other (in most cases MPB- 
NGC is the slightly more successful one), the performance differences between 
the considered two algorithms can be completely neglected for one quarter of 
the examples, and for one third of the examples MPBNGC beats the reduced 
algorithm clearly. MPBNGG was not able to terminate successfully for many 
of the difficult examples in particular for N G {60,80,100} despite signifi¬ 
cantly longer running times as it can be seen in Figure [T2] (in additional test 
runs with a softer termination criterion MPBNGG did terminate for approxi¬ 
mately half of the difficult examples, but the quality of the obtained minimizers 
was not comparable with the corresponding x produced by the reduced algo¬ 
rithm, while for the comparisons presented here this quality is comparable) 
and therefore the reduced algorithm is superior to MPBNGG in 88 percent 
of these examples. Furthermore, for five percent of the examples one of these 
two solvers has small advantages over the other, the performance differences 
between the considered two algorithms can be completely neglected for fur¬ 
ther five percent of the examples, and for the remaining two percent of the 
examples MPBNGG beats the reduced algorithm clearly. 

For the easy examples the reduced algorithm is superior to SolvOpt in 
thirty percent of the examples, for fifteen percent of the examples one of these 
two solvers has small advantages over the other, the performance differences 
between the considered two algorithms can be completely neglected for fur¬ 
ther fifteen percent of the examples, and in the remaining forty percent of 
the examples SolvOpt beats the reduced algorithm clearly. For the difficult 
examples the reduced algorithm is superior to SolvOpt in a bit more than half 
of the examples (including many examples with N G (80,100}), for twenty 
percent of the examples one of these two solvers has small advantages over the 
other, the performance differences between the considered two algorithms can 
be completely neglected for ten percent of the examples, and in the remaining 
(a bit less than) twenty percent of the examples SolvOpt beats the reduced 
algorithm clearly. In particular note that only very few Fj are active at the 
points which SolvOpt found at termination for the easy examples (in compari¬ 
son to both the reduced algorithm and MPBNGG), which might indicate that 
SolvOpt has some problems coming very close to the boundary. Although this 
behavior improves for the difficult examples, there still remains a clear gap in 
the number of active Fj between SolvOpt and the other two solvers. 

We want to emphasize the reduced algorithm was the only solver that ter¬ 
minated for all higher dimensional examples successfully, i.e. with a stationary 
point that is sufficiently accurate. Moreover, the solving times of the reduced 
algorithm are quite stable over all dimensions N G (20,40,60,80,100}. 


Remark 1 Since MOSEK supports multiple CPUs in particular for solving 
QCQPs (cf. Andersen [i|, p.l52, 8.1.4 Using multiple CPU’s]), we may expect 
faster solving times for the reduced algorithm on such a system in particular 
for higher dimensional problems. Nevertheless, we have not been able to test 
this yet. 
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We also expect a significant improvement of the full algorithm if a QCQP- 
solver is used which exploits the special structure of the QCQP (0]). 


4^.4.1 Easy examples with N/2 constraint components 

We summarize the investigations of the results of the easy examples with 
m2 := which can be found in Table 9 in Fendl & Schichl Q and which 
are visualized in Figure 30 in Fendl & SCHiCHL Q. 


Reduced algorithm vs. MPBNGC MPBNGC does not satisfy its termination 
criterion for one example within the given number of iterations and function 
evaluations. For the remaining 99 examples we obtain that the reduced algo¬ 
rithm is significantly better in 8 examples, better in 5 examples and a bit better 
in 6 examples in comparison with MPBNGC, while MPBNGC is signihcantly 
better in 18 examples, better in 17 examples a bit better in 18 examples in 
comparison with the reduced algorithm, and in 27 examples the solving times 
of both algorithms do not differ significantly. 


Reduced algorithm vs. SolvOpt SolvOpt does not satisfy its termination cri¬ 
terion for 18 examples within the given number of iterations and function 
evaluations. For the remaining 82 examples we obtain that the reduced al¬ 
gorithm is significantly better in 4 examples, better in 7 examples and a bit 
better in 6 examples in comparison with SolvOpt, while SolvOpt is signifi¬ 
cantly better in 14 examples, better in 25 examples and a bit better in 11 
examples in comparison with the reduced algorithm, and in 15 examples the 
solving times of both algorithms do not differ significantly. 


4.4-2 Easy examples with N constraint components 

We summarize the investigations of the results of the easy examples with 
m2 := N, which can be found in Table 10 in Fendl & Schichl @ and which 
are visualized in Figure 31 in Fendl & Schichl 


Reduced algorithm vs. MPBNGC MPBNGC does not satisfy its termination 
criterion for two examples within the given number of iterations and function 
evaluations. For the remaining 98 examples we obtain that the reduced al¬ 
gorithm is significantly better in 5 examples, better in 5 examples and a bit 
better in 8 examples in comparison with MPBNGC, while MPBNGC is sig¬ 
nificantly better in 8 examples, better in 26 examples and a bit better in 26 
examples in comparison with the reduced algorithm, and in 20 examples the 
solving times of both algorithms do not differ significantly. 



Title Suppressed Due to Excessive Length 


35 


Reduced algorithm vs. SolvOpt SolvOpt does not satisfy its termination cri¬ 
terion for 11 examples within the given number of iterations and function 
evaluations. For the remaining 89 examples we obtain that the reduced algo¬ 
rithm is significantly better in 5 examples, better in 11 examples and a bit 
better in 10 examples in comparison with SolvOpt, while SolvOpt is signif¬ 
icantly better in 16 examples, better in 21 examples and a bit better in 11 
examples in comparison with the reduced algorithm, and in 15 examples the 
solving times of both algorithms do not differ significantly. 

Difficult examples with N/2 constraint components 

We summarize the investigations of the results of the difficult examples with 
TO 2 := Y, which can be found in Table 11 in Fendl & SCHICHL @ and which 
are visualized in Figure 32 in Fendl & SCHiCHL 

Reduced algorithm vs. MPBNGC MPBNGC does not satisfy its termination 
criterion for 73 examples within the given number of iterations and function 
evaluations. For the remaining 27 examples we obtain that the reduced al¬ 
gorithm is significantly better in 9 examples, better in 2 examples and a bit 
better in 3 examples in comparison with MPBNGC, while MPBNGC is better 
in 4 examples and a bit better in 5 examples in comparison with the reduced 
algorithm, and in 4 examples the solving times of both algorithms do not differ 
significantly. 

Reduced algorithm vs. SolvOpt SolvOpt does not satisfy its termination crite¬ 
rion for 3 examples within the given number of iterations and function evalua¬ 
tions. For the remaining 97 examples we obtain that the reduced algorithm is 
significantly better in 23 examples, better in 28 examples and a bit better in 
15 examples in comparison with SolvOpt, while SolvOpt is significantly better 
in 4 examples, better in 16 examples and a bit better in 3 examples in com¬ 
parison with the reduced algorithm, and in 8 examples the solving times of 
both algorithms do not differ significantly. 

4-4-4 Difficult examples with N constraint components 

We summarize the investigations of the results of the difficult examples with 
TO 2 := N, which can be found in Table 12 in Fendl & Schichl @ and which 
are visualized in Figure 33 in Fendl & Schichl Q. 

Reduced algorithm vs. MPBNGC MPBNGC does not satisfy its termination 
criterion for 78 examples within the given number of iterations and function 
evaluations. For the remaining 22 examples we obtain that the reduced algo¬ 
rithm is significantly better in 10 examples, better in 4 examples and a bit 
better in 1 example in comparison with MPBNGC, while MPBNGC is better 
in 1 example and a bit better in 1 example in comparison with the reduced 
algorithm, and in 5 examples the solving times of both algorithms do not differ 
significantly. 
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Reduced algorithm vs. SolvOpt For our 100 examples we obtain that the re¬ 
duced algorithm is significantly better in 20 examples, better in 34 examples 
and a bit better in 7 examples in comparison with SolvOpt, while SolvOpt is 
significantly better in 5 examples, better in 8 examples and a bit better in 11 
examples in comparison with the reduced algorithm, and in 15 examples the 
solving times of both algorithms do not differ significantly. 


5 Conclusion 

In this paper we investigated numerical aspects of the feasible second order 
bundle algorithm for nonsmooth, nonconvex optimization problems with in¬ 
equality constraints. Since one of the main characteristics of this method is 
that the search direction is determined by solving a convex QCQP, we in¬ 
vestigated certain versions of the search direction problem and we justified 
the version chosen by us numerically by comparing the results of different 
solvers for the computation of the search direction. Furthermore, we made 
comparisons between the test results of our implementation of the second or¬ 
der bundle algorithm, MPBNGC by Makela [2^ and SolvOpt by Kappel 

6 Kuntsevich |i4 f or some examples of the Hock-Schittkowski collection 
by SCHITTKOWSKI [SSL |34| and for custom examples that arise in the context 
of finding exclusion boxes for quadratic CSPs, where for both of these types 
of examples we were able to achieve good results with respect to the number 
of evaluations of function dependent data, as well as for higher dimensional 
piecewise quadratic examples, in which our implementation achieved good re¬ 
sults in comparison with the other solvers in particular in the case that many 
constraint components were active at the solution. Summarizing the results it 
can be seen that the the SQP-like algorithm tends to compare the better the 
higher the dimension of the problem and the more difficult the nonsmoothness 
around the optimal point are. 
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